Search for: All records

The recent emergence of powerful Vision-Language models (VLMs) has significantly improved image captioning. Some of these models are extended to caption videos as well. However, their capabilities to understand complex scenes are limited, and the descriptions they provide for scenes tend to be overly verbose and focused on the superficial appearance of objects. Scene descriptions, especially in movies, require a deeper contextual understanding unlike general-purpose video captioning. To address this challenge, we propose a model, CALVIN, a specialized video LLM that leverages previous movie context to generate fully “contextual” scene descriptions. To achieve this, we train our model on a suite of tasks that integrate both image-based question-answering and video captioning within a unified framework, before applying instruction tuning to refine the model’s ability to provide scene captions. Lastly, we observe that our model responds well to prompt engineering and few-shot in-context learning techniques, enabling the user to adapt it to any new movie with very little additional annotation. 

This paper proposes solutions to detecting and mitigating the blatant replication and memorization of data used to train text-to-image generators, especially Stable Diffusion. The potential for diffusion models to reproduce copyrighted or private images without user knowledge poses significant ethical and legal challenges. For lawmakers, this highlights the need for clear guidelines and regulations around the use of such models, especially in commercial applications.